[OCTRL-1091] Integrate k8s client into ECS core#833
Conversation
29df8e4 to
d161264
Compare
justonedev1
changed the title
justonedev1
changed the title
knopers8
left a comment
knopers8
left a comment
There was a problem hiding this comment.
Thanks, i have comments mostly to naming, while the logic and approach seems OK for me.
task/manager.go becomes less clean due to placing kubernetes stuff there, but I wouldn't worry too much with creating a common abstraction for k8s + mesos, since this code is subject to further rewriting or starting it from scratch without mesos in the first place.
| taskman.ackKilledTasks = safeacks.NewAcks() | ||
| taskman.k8sEnvs = newK8sEnvRegistry() | ||
| if k8sClient, k8sErr := newK8sClientFromViper(); k8sErr != nil { | ||
| log.WithError(k8sErr).Warn("K8s client init failed, K8s tasks disabled") |
There was a problem hiding this comment.
perhaps warn/devel for the time being, so no one panics when using AliECS in a "classic" setup?
| 3. Inside the cluster, the **task-manager** and **environment-manager** operators (see `control-operator/cmd/`) watch these CRDs and reconcile the desired state — scheduling Pods, driving OCC gRPC state transitions, and writing status back into the CRD. | ||
| 4. ECS watches the CRDs via the Kubernetes Watch API and reacts to status changes exactly as it would to Mesos task updates, keeping the rest of the ECS state machine intact. | ||
|
|
||
| This design lets ECS reuse its existing environment lifecycle, configuration generation, and monitoring logic while outsourcing Pod scheduling and OCC communication to the in-cluster operators. The relevant code can be found in manager.go and managerk8s.go. |
There was a problem hiding this comment.
We might need to be more specific, as there is task/manager.go and environment/manager.go
| // Split K8s descriptors out before the Mesos retry loop. | ||
| var k8sDescriptors Descriptors | ||
| { | ||
| mesos := make(Descriptors, 0, len(tasksToRun)) |
There was a problem hiding this comment.
| mesos := make(Descriptors, 0, len(tasksToRun)) | |
| mesosDescriptors := make(Descriptors, 0, len(tasksToRun)) |
| m.roster.append(e.task) | ||
| } | ||
|
|
||
| envCRDName, err := m.createK8sEnvironmentCRD(ctx, envId, nodeToRefs) |
There was a problem hiding this comment.
I might be mistaken, but isn't it a custom object that we are creating, not Custom Resource Definition? My understanding is that a CRD defines what fields our custom objects should have at the new custom resource/endpoint.
| envCRDName, err := m.createK8sEnvironmentCRD(ctx, envId, nodeToRefs) | |
| envCRDName, err := m.createK8sEnvironmentObject(ctx, envId, nodeToRefs) |
So if I am right, any occurence of "CRD" should be replaced here with "Object" or removed.
| log.WithField("partition", envCRDName). | ||
| WithField("crd", envCRDName).Debug("received some environment update") | ||
| if env.Name == envCRDName { | ||
| log.WithField("crd", envCRDName). |
There was a problem hiding this comment.
| log.WithField("crd", envCRDName). | |
| log.WithField("crd", envCRDName). | |
| WithField("partition", envCRDName). |
2 participants
This is actual implementation of the ECS Kubernetes bridge.
Before merging it needs the OCTRL-1090
The goal of this PR was to take existing code that deploys Mesos tasks and adjusts it for the kubernetes client usage, basically linking CRDs defined and managed by managers in control-operator to the existing ECS task infrastructure and classes. The code from managerk8s.go was inspired roughly accroding to this table (generated by llm, but I went through it):
managerk8s.gocreateK8sTaskEntries— InboundChannels loop, IPC endpoint,::globalaliasmakeTaskForMesosResources— same loop, IPC/TCP handling, global aliasscheduler.go:1346buildK8sNodeTaskRefs—O2_ROLE/O2_SYSTEMenv var injection loopmakeTaskForMesosResources—fillEnvDefaultloop for the same two varsscheduler.go:1458buildK8sNodeTaskRefs— ArgsCLI trimming/filteringmakeTaskForMesosResources—cmd.Argumentshandlingscheduler.go:1437newTaskForKubernetesnewTaskForMesosOffermanager.go:169deployKubernetesTasks— overall flowacquireTasks— Mesos deployment blockmanager.go:550configureK8sTasks—BuildPropertyMapper task, global bind mapconfigureTasks— bind map building +BuildPropertyMapsmanager.go:804transitionAndWaitK8sEnvState— send transition, block until state reachedtransitionTasks— send Mesos command, block on responsemanager.go:901killK8sTaskdoKillTaskmanager.go:1238killK8sEnvironmentreleaseTasksmanager.go:733process— route watch event to roster task (called fromwatchK8sTasks)updateTaskState/updateTaskStatusmanager.go:1071watchK8sTasks— retry loop on watcher failurescheduler.gonodeNameFromDescriptor— scan constraints formachine_idSatisfy(targetConstraints)scheduler.go:743k8sEnvRegistry,createK8sEnvironmentCRD,waitForK8sEnvState,cmdEnvToK8sEnvVarsIt contains multiple changes:
controlmodes to distinguish between Kubernetes and Mesos tasksreadout,stfbuilder-senderoutputandstfsender